<!doctype html>
<html lang="zh-CN"><head><script src="/livereload.js?mindelay=10&amp;v=2&amp;port=1313&amp;path=livereload" data-no-instant defer></script>
    <title>薛尧的博客</title>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <meta name="description" content="" />

    
    
    
    <link rel="stylesheet" href="../../../css/theme.min.css">

    
    
    
    
    <link rel="stylesheet" href="../../../css/custom.min.css">
    

    
</head>
<body>
        <div id="content" class="mx-auto"><header class="container mt-sm-5 mt-4 mb-4 mt-xs-1">
    <div class="row">
        
        <div class="col-sm-4 col-12 text-sm-right text-center pt-sm-4">
            <a href="../../../" class="text-decoration-none">
                <img id="home-image" class="rounded-circle"
                    
                        src="https://ueyao.github.io/image-hosting/blog/avatar.png"
                    
                />
            </a>
        </div>
        <div class="col-sm-8 col-12 text-sm-left text-center">
        
            <h2 class="m-0 mb-2 mt-4">
                <a href="../../../" class="text-decoration-none">
                    
                        薛尧
                    
                </a>
            </h2>
            <p class="text-muted mb-1">
                
                    Java Developer | Short Video Creator
                
            </p>
            <ul id="nav-links" class="list-inline mb-2">
                
                
                    <li class="list-inline-item">
                        <a class="badge badge-white " href="../../../about/" title="关于">关于</a>
                    </li>
                
                    <li class="list-inline-item">
                        <a class="badge badge-white " href="../../../post/" title="文章">文章</a>
                    </li>
                
                    <li class="list-inline-item">
                        <a class="badge badge-white " href="../../../categories/" title="分类">分类</a>
                    </li>
                
            </ul>
            <ul id="nav-social" class="list-inline">
                
                    <li class="list-inline-item mr-3">
                        <a href="http://github.com/flowstone" target="_blank">
                            <i class="fab fa-github fa-1x text-muted"></i>
                        </a>
                    </li>
                
                    <li class="list-inline-item mr-3">
                        <a href="mailto:xueyao.me#gmail.com" target="_blank">
                            <i class="fas fa-at fa-1x text-muted"></i>
                        </a>
                    </li>
                
            </ul>
        </div>
    </div>
    <hr />
</header>
<div class="container">
    <div class="pl-sm-2">
        <div class="mb-3">
            <h3 class="mb-0">Docker进阶-容器监控cAdvisor&#43;InfluxDB&#43;Granfana</h3>
            
            <small class="text-muted">发布于 2018-09-07</small>
        </div>

        <article>
            <h2 id="概述">概述</h2>
<p>前面文章介绍使用docker compose组合应用并利用scale快速对容器进行扩容。</p>
<p>由于docker compose启动的服务都在同一台宿主机上，对于一个宿主机上运行多个容器应用时，容器的运行情况如：CPU使用率、内存使用率、网络状态、磁盘空间等一系列随时间变化的时序数据信息，都是需要去了解，因此监控是必须的。</p>
<h2 id="容器监控方案选择">容器监控方案选择</h2>
<p>对于容器的监控方案可谓多种多样，本身自带命令docker stats命令，Scout,Data Dog,Sysdig Cloud,Sensu Monitoring Framework,CAdvisor等。</p>
<p>通过docker stats命令可以很方便的看到当前宿主机上所有容器的CPU,内存以及网络流量等数据。但是docker stats命令的缺点就是统计的只是当前宿主机的所有容器，而获取的监控数据是实时的，没有地方存储,也没有报警功能。</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker stats
</span></span></code></pre></div><p>而Scout、Sysdig Cloud、Data Dog虽然都提供了较完善的服务，但是它们都是托管的服务而且都收费，Sensu Monitoring Framework集成度较高，也免费，但是部署过于复杂。综合考虑，我们选择了CAdvisor做容器监控工具。</p>
<p>CAdvisor谷歌出品，优点是开源产品，监控指标齐全，部署方便，而且有官方的docker镜像。缺点是集成度不高，默认只在本地保存2分钟数据。不过可以加上InfluxDB存储数据，对接Grafana展示图表，比较便利地搭建好了容器监控系统，数据收集和图表展示效果良好，对系统性能也几乎没有什么影响。</p>
<h2 id="cadvisor">CAdvisor</h2>
<p>CAdvisor是一个容器资源监控工具,包括容器的内存,CPU,网络IO,磁盘IO等监控,同时提供了一个WEB页面用于查看容器的实时运行状态。CAdvisor默认存储2分钟的数据,而且只是针对单物理机。不过，CAdvisor提供了很多数据集成接口,支持InfluxDB,Redis,Kafka,Elasticsearch等集成,可以加上对应配置将监控数据发往这些数据库存储起来。</p>
<p>CAdvisor功能主要有两点：</p>
<ul>
<li>
<p>展示Host和容器两个层次的监控数据。</p>
</li>
<li>
<p>展示历史变化数据。</p>
</li>
</ul>
<h2 id="influxdb">InfluxDB</h2>
<p>InfluxDB是用Go语言编写的一个开源分布式时序、事件和指标数据库,无需外部依赖。</p>
<p>前面说到,CAdvisor默认只在本机保存最近2分钟的数据，为了持久化存储数据和统一收集展示监控数据，需要将数据存储到InfluxDB中。InfluxDB是一个时序数据库,专门用于存储时序相关数据，很适合存储CAdvisor的数据。而且，CAdvisor本身已经提供了InfluxDB的集成方法，丰启动容器时指定配置即可。</p>
<p>InfluxDB主要功能:</p>
<ul>
<li>
<p>基于时间序列,支持与时间有关的相关函数(如最大、最小、求和等);</p>
</li>
<li>
<p>可度量性:你可以实时对大量数据进行计算;</p>
</li>
<li>
<p>基于事件:它支持任意的事件数据;</p>
</li>
</ul>
<p>InfluxDB主要特点:</p>
<ul>
<li>
<p>无结构(无模式);</p>
</li>
<li>
<p>可以是任意数量的列;</p>
</li>
<li>
<p>可拓展的;</p>
</li>
<li>
<p>支持min,max,sum,count,mean,median等一系列函数,方便统计;</p>
</li>
<li>
<p>原生的HTTP支持,内置HTTP API;</p>
</li>
<li>
<p>强大的类SQL语法;</p>
</li>
<li>
<p>自带管理界面,方便使用</p>
</li>
</ul>
<h2 id="granfana">Granfana</h2>
<p>Grafana是一个开源的数据监控分析可视化平台,支持多种数据源配置(支持的数据源包括InfluxDB,MySQL,Elasticsearch,OpenTSDB,Graphite等)和丰富的插件及模板功能,支持图表权限控制和报警。</p>
<p>Grafan主要特性:</p>
<ul>
<li>
<p>灵活丰富的图形化选项</p>
</li>
<li>
<p>可以混合多种风格</p>
</li>
<li>
<p>支持白天和夜间模式</p>
</li>
<li>
<p>多个数据源</p>
</li>
</ul>
<h2 id="cadvisorinfluxdbgranfana">CAdvisor+InfluxDB+Granfana</h2>
<p>CAdvisor:负责收集容器的随时间变化的数据</p>
<p>InfluxDB:负责存储时序数据</p>
<p>Grafana:负责分析和展示时序数据</p>
<h2 id="安装部署">安装部署</h2>
<h3 id="部署influxdb服务">部署InfluxDB服务</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker run -d --name influxdb -p 8086:8086 <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>-v /data/influxdb:/var/lib/influxdb <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>--hostname<span style="color:#f92672">=</span>influxdb <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>influxdb
</span></span></code></pre></div><p>说明：</p>
<ul>
<li>
<p>&ndash;name:启运容器分配名字influxdb</p>
</li>
<li>
<p>-p:映射端口，8083端口为influxdb管理端口，8086端口是influxdb的数据端口</p>
</li>
<li>
<p>-v:挂载数据 -d:后台运行此容器</p>
</li>
<li>
<p>influxdb:通过这个容器来运行的，默认会在docker官方仓库pull下来</p>
</li>
</ul>
<p>1、浏览器访问influxdb的管理端，http://ip:8083</p>
<p>2、创建test的数据库与root用户，这个用于后期配granfa</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec it influxdb influx
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>CREATE DATABASE <span style="color:#e6db74">&#34;test&#34;</span>
</span></span><span style="display:flex;"><span>CREATE USER <span style="color:#e6db74">&#34;root&#34;</span> WITH PASSWORD <span style="color:#e6db74">&#39;root&#39;</span> WITH ALL PRIVILEGES
</span></span></code></pre></div><h3 id="部署cadvisor服务">部署CAdvisor服务</h3>
<p>谷歌的cadvisor可以用于收集Docker容器的时序信息，包括容器运行过程中的资源使用情况和性能数据。</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker run --volume<span style="color:#f92672">=</span>/:rootfs:ro --volume<span style="color:#f92672">=</span>/var/run:/var/run:ro<span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>--volume<span style="color:#f92672">=</span>/sys:/sys:ro --volume<span style="color:#f92672">=</span>/var/lib/docker/:/var/lib/docker:ro<span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>--volume<span style="color:#f92672">=</span>/dev/disk/:/dev/disk:ro --volume<span style="color:#f92672">=</span>/cgroup:/cgroup:ro <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>--publish<span style="color:#f92672">=</span>8080:8080 --detach<span style="color:#f92672">=</span>true --privileged<span style="color:#f92672">=</span>true --name<span style="color:#f92672">=</span>cadvisor <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>google/cadvisor:latest<span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>-storage_driver<span style="color:#f92672">=</span>influxdb-storage_driver_host<span style="color:#f92672">=</span>influxdb:8086<span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>-storage_driver_db<span style="color:#f92672">=</span>test-storage_driver_user<span style="color:#f92672">=</span>root<span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>-storage_driver_password<span style="color:#f92672">=</span>root
</span></span></code></pre></div><p>说明：</p>
<ul>
<li>
<p>-d:后台运行此容器</p>
</li>
<li>
<p>&ndash;name:启动容器分配名字cadvisor</p>
</li>
<li>
<p>-p:映射端口8080</p>
</li>
<li>
<p>&ndash;mount:把宿主机的相关目录绑定到容器中，这些目录都是cadvisor需要采集的目录文件 和监控内容</p>
</li>
<li>
<p>-storage_driver:需要指定cadvisor的存储驱动、数据库主机、数据库名</p>
</li>
</ul>
<p>通过ip:8080端口访问测试一下</p>
<h3 id="部署granfana服务">部署Granfana服务</h3>
<p>grafana是一款开源的时序数据分析工具，而且界面专业简单。</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker run -d -p 3000:3000 <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>-v /data/grafana:/var/lib/grafana <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>--link<span style="color:#f92672">=</span>influxdb:influxdb<span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span>--name grafana  grafana/grafana
</span></span></code></pre></div><p>注意：如果使用-v把数据挂载出来会出现容器启动失败，错误为mkdir:cannot create directory &lsquo;/var/lib/grafana/plugins&rsquo;:Permission denied,此时可以使用docker run &ndash;entrypoint &ldquo;id&rdquo; grafana/grafana得到uid,gid,groups(默认是472);</p>
<p>接着使用chown -R 472:472 /data/grafana/修改目录权限就可以启动成功。</p>
<p>通过ip:3000端口来访问grafana的web服务，第一次访问需要修改admin用户密码，默认用户名/密码为admin/admin</p>
<h3 id="docker-compose快速部署">docker-compose快速部署</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">version</span>: <span style="color:#e6db74">&#39;3.1&#39;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">volumes</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">grafana_data</span>: {}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">services</span>:
</span></span><span style="display:flex;"><span> <span style="color:#f92672">influxdb</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">image</span>: <span style="color:#ae81ff">tutum/influxdb:0.9</span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">#image: tutum/influxdb</span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">#image: influxdb</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">restart</span>: <span style="color:#ae81ff">always</span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">#user: </span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">environment</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">PRE_CREATE_DB=cadvisor</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ports</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#e6db74">&#34;8083:8083&#34;</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#e6db74">&#34;8086:8086&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">expose</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#e6db74">&#34;8090&#34;</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#e6db74">&#34;8099&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">volumes</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">./data/influxdb:/data</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#f92672">cadvisor</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">#image: google/cadvisor:v0.29.0</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">image</span>: <span style="color:#ae81ff">google/cadvisor</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">links</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">influxdb:influxsrv</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">command</span>: -<span style="color:#ae81ff">storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_host=influxsrv:8086</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">restart</span>: <span style="color:#ae81ff">always</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ports</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#e6db74">&#34;8080:8080&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">volumes</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">/:/rootfs:ro</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">/var/run:/var/run:rw</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">/sys:/sys:ro</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">/var/lib/docker/:/var/lib/docker:ro</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#f92672">grafana</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">#image: grafana/grafana:2.6.0</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">user</span>: <span style="color:#e6db74">&#34;104&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">image</span>: <span style="color:#ae81ff">grafana/grafana</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">user</span>: <span style="color:#e6db74">&#34;104&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">#user: &#34;472&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">restart</span>: <span style="color:#ae81ff">always</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">links</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">influxdb:influxsrv</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">ports</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#e6db74">&#34;3000:3000&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">volumes</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">grafana_data:/var/lib/grafana</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">environment</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">HTTP_USER=admin</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">HTTP_PASS=admin</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">INFLUXDB_HOST=influxsrv</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">INFLUXDB_PORT=8086</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">INFLUXDB_NAME=cadvisor</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">INFLUXDB_USER=root</span>
</span></span><span style="display:flex;"><span>    - <span style="color:#ae81ff">INFLUXDB_PASS=root</span>
</span></span></code></pre></div><p>启动docker-compose文件</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose up
</span></span></code></pre></div><p>查看三个服务容器是否启动</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-01.png" alt=""></p>
<p>浏览grafana服务，http://ip:3000</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-02.png" alt=""></p>
<p>登陆grafana服务</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-03.png" alt=""></p>
<p>浏览influxdb服务，http://ip:8083/</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-04.png" alt=""></p>
<h3 id="配置grafana工具">配置grafana工具</h3>
<p>1、配置数据源</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-05.png" alt=""></p>
<p>选择influxdb数据源</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-06.png" alt=""></p>
<p>具体配置如下</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-07.png" alt=""></p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-08.png" alt=""></p>
<p>配置完成后，如下图</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-09.png" alt=""></p>
<p>2、配置面板</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-10.png" alt=""></p>
<p>选择面板类型</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-11.png" alt=""></p>
<p>选择图形</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-13.png" alt=""></p>
<p>配置查询条件</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-14.png" alt=""></p>
<p>配置完后，效果如下图</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-15.png" alt=""></p>
<p>grafana支持查询条件，可以从influxdb中了解</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-16.png" alt=""></p>
<p>浏览cAdvisor服务，http://ip:8080/</p>
<p><img src="https://ueyao.github.io/image-hosting/blog/2019/8/docker-monitor-17.png" alt=""></p>

        </article>
    </div>

    <div id="disqus_thread"></div>
<script>
    window.disqus_config = function () {
    
    
    
    };
    (function() {
        if (["localhost", "127.0.0.1"].indexOf(window.location.hostname) != -1) {
            document.getElementById('disqus_thread').innerHTML = 'Disqus comments not available by default when the website is previewed locally.';
            return;
        }
        var d = document, s = d.createElement('script'); s.async = true;
        s.src = '//' + "xie-yao-de-bo-ke" + '.disqus.com/embed.js';
        s.setAttribute('data-timestamp', +new Date());
        (d.head || d.body).appendChild(s);
    })();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
<a href="https://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>

            </div>
        </div><footer class="text-center pb-1">
    <small class="text-muted">
        
        &copy; Copyright 2024
        
        |
        <a href="https://beian.miit.gov.cn/" target="_blank">苏ICP备14012079号</a>
        <br />
        由 <a href="https://gohugo.io/" target="_blank">Hugo</a> 强力驱动
        | 主题 <a href="https://github.com/austingebauer/devise" target="_blank">Devise</a>
        <br />

    </small>
</footer></body>
</html>
